1. Field of the Invention
This invention relates generally to superresolution electro-optic imaging systems, including the “end-to-end” design of such systems.
2. Description of the Related Art
Electro-optic imaging systems typically include an optical subsystem (e.g., a lens assembly), an electronic detector subsystem (e.g., CCD detector array) and a digital image processing subsystem (e.g., typically implemented in dedicated chips or software). In most electro-optical imaging systems, the spatial sampling rate of the photodetector is well below the diffraction limit of the optical subsystem. In current technology, the smallest pixel dimensions (i.e., pixel-to-pixel pitch) are typically on the order of 3 to 4 microns. The corresponding Nyquist rate associated with such pixel dimensions are between 125 and 166 line pairs per millimeter (lp/mm). It is not uncommon to have optical subsystems with an F# as low as 3 or 4. Given that the diffraction limit is given by 1/(λ F#), diffraction limited optical subsystems can pass image content with spatial frequencies as high as 500 lp/mm in the visible spectrum.
FIG. 1 shows an example of a modulation transfer function (MTF) 110 for an F/4.5 diffraction-limited optical subsystem, the MTF 120 for a 100 percent fill factor 15 micron pitch pixel, and the cumulative MTF 130 for the optical subsystem and detector together. For convenience, the MTF for the optical subsystem will be referred to as the optical MTF 110, the MTF for the detector subsystem as the detector MTF 120, and the combined MTF as the imaging MTF 130. The imaging MTF is the product of the optical MTF and the imaging MTF. Also shown is the Nyquist rate for the detector subsystem which is 33 lp/mm in this example. The Nyquist sample rate will also be referred to as the detector sampling frequency. The box 140 indicates the MTF region up to the Nyquist rate. There is a significant fraction of the imaging MTF 130 that lies outside the sampling band 140 (i.e., at frequencies higher than the sampling frequency). Consequently, this electro-optical imaging system has the potential to pass image content with spatial frequencies above the Nyquist rate.
In theory, the image content at higher frequencies could be captured by reducing the pitch of the detector array, thus increasing the detector sampling frequency. However, the ability to shrink pixel dimensions is limited. As pixel dimensions shrink, the dynamic range and signal to noise ratio (SNR) of pixels degrade.
Returning to FIG. 1, when spatial frequency information above the Nyquist rate is sampled, the final image may contain aliasing artifacts such as moirépatterns. The effect of aliasing is even more pronounced in color systems using a single photodetector. In such cases, the Bayer pattern reduces the Nyquist rate by a factor of two further exacerbating the problem of aliasing. Researchers have developed a variety of techniques to eliminate aliasing artifacts. To some degree or another, these approaches typically involve some form of an optical low pass filter that effectively destroys the information content above the Nyquist rate. For instance, Kodak sells an optically transparent plate that is placed directly in front of the detector. The plate has randomly placed particles which introduce random phase errors. This effectively blurs the optical image, thus reducing the content at frequencies above the Nyquist rate and reducing the effects of aliasing.
In another approach, the image content is replicated in a color-dependent fashion using the spatial shifting property of a birefringent plate. The birefringent plate replicates the point spread function of the optical subsystem but shifted with respect to the original point spread function. The cumulative point spread function created by the original and its shifted versions can span one or two pixel widths. This replication effectively blurs the optical image to reduce frequency information above the Nyquist rate. However, such optical low pass filters often are wavelength dependent.
In yet another approach, CDM Optics of Boulder, Colo. developed a specially designed phase plate that is placed at the aperture of the optical subsystem in order to encode the incoming wavefront in a particular way. Digital image processing is used later to reverse the encoding introduced by the phase plate and retrieve certain image content. However, the CDM approach appears to work for only certain types of artifacts and it can produce overly smooth images.
Superresolution is a different approach that tries to make use of the aliased information rather than suppress it. Superresolution takes a collection of lower resolution images that contain aliased image content and produces a single image or set of images with higher resolution. For example, in a conventional superresolution system, the optical subsystem might produce a diffraction-limited image that is captured by the detector subsystem. A number of shifted versions of the image may be captured and then combined to form a higher resolution image. However, even though the superresolution processing effectively increases the sampling frequency, many high quality optical subsystems have imaging MTFs that still contains significant energy in frequencies above the effective superresolved sampling frequency. This can continue to cause aliasing artifacts.
Thus, there is a need for approaches that can take advantage of image content that is above the detector sampling frequency and/or that reduces aliasing effects, but in a manner that overcomes some or all of the above drawbacks.